Santayana is challenging the assumptions people make in every day life when assigning emotions or categories to other people.  The first line questions what you have that makes you "rich".  When you label someone as "poor", then such a comparison must be made against some level of monetary value.
1 5 1