You can meaningfully add or subtract degrees to a temperature reading, but you can't meaningfully add two readings together (unless you're averaging them!). There's a subtlety here: A temperature difference is actually a different data type than a temperature. Dates and times have similar issues: You can add 2 hours to 3 hours, and you can add 2 hours to 3 o'clock, but you can't add 2 o'clock to 3 o'clock.
A better litmus test is whether you can subtract values. The difference between two times, or two dates, or two temperature readings is something that is meaningful and commonly used in everyday life. The difference between two ZIP codes is nonsensical.
Nitpicking, but that seems like a great case for them being numbers. You will never average credit cards or zip codes. Averaging times is meaningful too (e.g. the mean arrival time).
I think op meant "adding" as generally doing math on them. Each quantity will have its own natural operations.
I see what you mean though, subtraction seems like a good heuristic by virtue of difference being mostly (always?) meaningful.
A better litmus test is whether you can subtract values. The difference between two times, or two dates, or two temperature readings is something that is meaningful and commonly used in everyday life. The difference between two ZIP codes is nonsensical.