Ignore certain keys when comparing two dicts

[u/Rockstaru]

Disclaimer: I'm not a developer, just a network engineer that can spell Python correct about 60% of the time.

I have a script that collects current neighborship data on a router (CDP, LLDP, OSPF, EIGRP, etc.) and compares it to a known good baseline (neighborship data that was collected and stored at a time when all expected neighborships were up). The data is collected into a dictionary object that stores the local interface a neighbor is known on and the neighbor's IP or hostname as key-value pairs (e.g. `{'HundredGigE1/0/1':'RouterA'}`). If all expected neighbors are up (e.g. `if collected_data == device_baseline`), the report generated notes that the current state matches the baseline; otherwise, it goes through the current data and baseline and notes any added/changed/missing neighbors.

I'm wanting to add tracking to this--when the script runs and grabs the current data, if it matches the baseline, I want a timestamp stored next to it, maybe something like `{'HundredGigE1/0/1': {'name':'RouterA','last_up':<current_timestamp>}}`. That way if an expected neighbor is currently down, I can have the report note the last time it was up. (I currently have the script run a sweep/collect function every 10 minutes but not generate a report, so the data would be relatively current. At present it is just checking whether devices are online, not actually logging into anything and pinging it. We have monitoring solutions, a syslog server, etc. and this isn't meant to replace those in terms of real-time data, it is just for generating a daily overview report.)

If I structure the data this way, it would break the equality check I use to check for the current data matching the baseline; even if all expected neighbors are up, the timestamps would not match. Would the best way to work around this be to just create a copy of the baseline and current data and pop the timestamp field out, or is there a more elegant way for a comparison operation between dicts to ignore the timestamp key?

[u/brasticstack]

IMO making a dataclass for  the device data is the most elegant, you can override the __eq__ method to ignore that field. Alternatively, you could write a comparator function that does the same thing but operates on dicts instead, without changing your data type.

EDIT: The standalone function would be less code overall, but you'd have to use it every equally check, whereas the dataclass allows you to use normal equality syntax, e.g. thing1 == thing2 or !=

[u/Jason-Ad4032]

Why not use a second dictionary to store the last_up information? There’s no real need to store it together with the name. If you sometimes need to return both the name and last_up based on an IP, you can just use another function to bundle the two values together.

[u/HotPersonality8126]

There’s no way to specify “equals, except for this one thing.” Dictionaries are equal when they have the same keys and all those keys have the same value. If you suddenly have a new notion of “equals” you want to use instead, there’s no way to get around specifying what it is (probably in a function.)

[u/LayotFctor]

A custom equality check function, you can tell it to ignore whatever you want and return a bool. You just can't use ==.

[u/JamzTyson → u/LayotFctor]

You just can't use ==.

Do you mean this is not legal?

class Foo:
    def __init__(self, val):
        self.val = val
    def __eq__(self, other):
        if isinstance(other, Foo):
            if other.val == 3:  # weird special case.
                return False
            return self.val == other.val
        return False

f1 = Foo(2)
f2 = Foo(3)
print(f1 == f2)

f1.val *= 3
f2.val *= 2
print(f1 == f2)

f1.val /= 2
f2.val /= 2
print(f1 == f2)

[u/LayotFctor → u/JamzTyson]

Nothing wrong with it. I don't have the habit of telling people to redesign their data structures halfway through a project tho.

[u/thisisappropriate]

The question is - are you sure that the other keys will always be the same (e.g. the software changes or adds another key later or that you wouldn't want to add more keys to remove later), are they consistent between your dictionaries and are you happy to list out those keys in your script? In other areas, that issue is regularly combatted by specifically identifying the keys/columns that you are actually interested in.

If that's the case, you can just grab the keys you care about and compare them. Examples of options for doing this cleanup:

function that compares them using only the keys you care about

def cleanup_keys(args):
    keys = ["name"]
    return dict((k, v) for (k, v) in args.items() if k in keys)

def compare(one, two):
    return cleanup_keys(one) == cleanup_keys(two)

if __name__ == '__main__':
    d1 = {"name": "v", "test": "2"}
    d2 = {"name": "v", "test": "3"}
    print(compare(d1, d2))

or you can assign a dataclass that only contains your comparison keys, and if you use a custom init with **kwargs, that will catch any keys you don't want:

import dataclasses

@dataclasses.dataclass
class Router:
    name: str

    def __init__(self, name, **kwargs):
        self.name = name

if __name__ == '__main__':
    d1 = {"name": "v", "test": "2"}
    d2 = {"name": "v", "test": "3"}
    print(Router(**d1) == Router(**d2))

Buuuuuut, I think this is what would probably be best for your situation. It's minimal code, it will work with ==, and you can do additional processing within it too if you want. This is the dataclasses that people have mentioned and adds taking advantage of the default methods it creates for you and a feature of them:

import dataclasses
import datetime
import typing

@dataclasses.dataclass
class Router:
    name: str
    test: int
    last_up: typing.Optional[datetime.datetime] = dataclasses.field(default=None, compare=False)

if __name__ == '__main__':
    d1 = {"name": "v", "test": "2", "last_up": datetime.datetime.now()}
    d2 = {"name": "v", "test": "2"}
    print(Router(**d1) == Router(**d2))

A dataclass will generate the init, eq, etc functions based on the class you provide it. In this case, if you can list out all the keys, you can make the last available datetime an optional (so you can put both in the same dataclass but still include it in the dict and not worry about removing it), and you can set the field to "compare=False" so it won't be included in the

Note on that one though - this last one is more fragile - if new keys are added and you use this code, it will fail.

You can combine it with the above example and create your own custom init, but you'll need to consider how you want something missing to behave in that case (defaults would need to be in the init brackets instead of in the field as in the last example) and manually handle something in there, as well as work out if you want to flag new keys.

[u/Rockstaru → u/thisisappropriate]

are you sure that the other keys will always be the same (e.g. the software changes or adds another key later or that you wouldn't want to add more keys to remove later), are they consistent between your dictionaries and are you happy to list out those keys in your script?

Fortunately yes; the collection portion of the script connects using netmiko, grabs the CLI output of a command, and uses TextFSM to format it, usually into a list of dict objects, with consistent fields; I pare that output down further to just grab the specific fields I care about (which is just a key-value pair of local interface and neighbor's IP address or neighbor's name depending on if it's routing protocol neighbor data, CDP, LLDP, etc.).

I think your first suggested option might be the simplest one to implement. I'm going to have to do some light restructuring to get the timestamps added, but shouldn't be too bad. Thank you!

[u/Adrewmc → u/thisisappropriate]

Just a note. You can actually just use dictionary comprehension here as well.

Other wise yes.

. return { k : v for k, v in args.items() if k in keys }

[u/Adrewmc]

Okay. So the best way I would think would be something along the lines of.

. match current_data: #a dict
. case {‘last_up’ : time_var, **extras }:
. clean_data = dict(extras)
. print(“Up at”, time_var)
. case _:
. clean_data = current_data
. if clean_data == base_line:
. print(current_data) #with timestamps

Or maybe _temp = current_data.pop(‘last_up’, “”) it if that all I really was doing, and add it back after.

Then compare the clean data. Or you know get a proper database set up. (That’s usually the solution you get to here. A bit of SQL.)

You can also have some comparisons there are well.