Should we aim towards what we value?

I am currently reading Derek Parfit: His Life and Thought, a collection of essays written by some of his companions who he crossed paths with throughout his life. The first essay, written by British philosopher of ethics Jonathan Dancy, contained an interesting passage which caught my attention.

The Repugnant Conclusion is a result in the theory of value: a large enough world at a very low average level of well-being will always be better than a much smaller world with a much higher average level of well-being. But we should bear in mind the relevance of this to decisions about what we have reason to do. Parfit himself came to accept Scanlon's buck-passing conception of value, according to which the fact that something is of value is the same fact as the fact that we have reasons of certain sorts (to protect, promote, ..., it). And his phrase (in Reasons and Persons) "better in the sense relevant to choice" reveals a nascent tendency towards what one might call a deontic conception of value, since "relevant to choice" seems to mean something like "relevant to what we ought to (decide to) do". But if we reject such conceptions of value, we have room to suggest that, though a repugnant world would be better than the present smaller world, that fact gives us no reason to prefer it, to work towards it if we can, and so on. After all, the fact that a child, if I were to have it, would have a good quality of life is no reason to have that child - though it might perhaps intensify (or act as an enabler for) any reasons I do have to have that child. By contrast, the fact that, if I had a child, that child's life would not be worth living may be-probably would be-a reason not to have that child. But this need not determine our answer in the positive case.

Dancy seems to be describing a possible solution for the repugnant conclusion. The repugnant conclusion describes a well-known problem in population ethics that deals with large groups of people. Imagine a world (called world #1) with 10 billion extremely happy people. Let's say another person comes into this world (making it world #2), who, although not perfectly happy, still has a great life which is clearly worth living. It would be hard to not describe world #2 as preferable to world #1, given that it contains another person with a great life. Now, imagine that more and more people get added to that world, all slightly less happy than the one before, until we arrive at an existence that is barely worth living. More and more people like this, who are just slightly more happy than unhappy, get born, so that the average happiness of the world declines to just slightly above parity. For every step, the new world would be preferable to the one before; however, the end result world #N seems clearly worse than world #1.

As I understand Dancy, following the theory he explains, would mean that one can claim that world #N is superior to world #1, but that we should not be aiming towards that world.
That seems wrong to me. If we do not think that we should work towards what we think is valuable, then what is value good for? If I claim that giving to the homeless is the most valuable thing in the world, but then spend all my money on luxury goods, does "giving to the homeless" hold any value for me in a meaningful sense?