Pydantic: generate unique lists

[WilliamDEdwards]

Since Pydantic v2, unique lists are no longer natively supported (x: list = Field(unique=True).

Therefore, I resorted to using a custom validator (inspired by pydantic/pydantic-core#296 (comment) and pydantic/pydantic-core#820 (comment)):

UniqueListItem = TypeVar("UniqueListItem", bound=Hashable)

def validate_unique_list(value: list[UniqueListItem]) -> list[UniqueListItem]:
    """Validate that list contains only unique items.

    Taken from https://github.com/pydantic/pydantic-core/pull/820#issuecomment-1670475909
    """
    if len(value) == len(set(value)):
        return value

    raise PydanticCustomError("unique_list", "List must be unique")

PydanticTypeUniqueList = Annotated[
    List[UniqueListItem],
    AfterValidator(validate_unique_list),
    Field(json_schema_extra={"uniqueItems": True}),
]

Because Polyfactory doesn't know about UniqueListItem, it generates duplicate values, causing flakey tests.

Normally, I would just add UniqueListItem to the provider map, but that's an option here: Polyfactory wouldn't know what the list item's types should be.

So, I think I need a way to call Polyfactory's value generation for list items, inside a custom wrapper that prevents duplicates.

Can anyone point me to the primitives that I can use (assuming they are public APIs, and invokable outside the build context)? I read through factories.pydantic_factory and base.pydantic_factory, but couldn't find the answer yet.

[adhtruong]

Hi @WilliamDEdwards !

I think the would need to UniqueListItem should be bound at this point so would need to override PydanticTypeUniqueList so the constraint could be applied at that level. Some of the logic is quite deep in functions so not super easy to override this way

Some alternative approaches would be

1. Change type definition to

PydanticTypeUniqueList = Annotated[
    List[UniqueListItem],
    AfterValidator(validate_unique_list),
    Field(json_schema_extra={"uniqueItems": True}),
    {'unique_items': True}
]

This adds some duplication but uses existing tested constraint parsing so that this is applied for polyfactory.

2. Override on a per field level with process_kwargs (and coverage) so this is made unique. E.g.

class MyFactory(ModelFactory[A]):
      @classmethod
      def process_kwargs(self, *args, **kwargs) -> dict:
                kwargs = super().process_kwargs()
                kwargs['my_field'] = list(set(kwargs['my_field'])
               return kwargs

Exact typing and usage may need to be adjusted especially if other constraints on field

[WilliamDEdwards]

Hi @adhtruong,

Thanks for the hint; alternative approach #1 works like a charm!

It should be noted that unique_items is deprecated. From https://docs.pydantic.dev/2.10/api/types/#pydantic.types.conlist:

The unique_items parameter is deprecated, use Set instead. See this issue for more details.

... so I hope Polyfactory won't consider this a remnant after its removal by Pydantic.

[WilliamDEdwards]

This does not work with unhashable types however:

tests/factories/_base.py:182: in produce
    model = cls.build(cls.get_defaults(), overrides)
tests/factories/_base.py:65: in build
    return pydantic_factory.build(**fields)
.venv/lib/python3.11/site-packages/polyfactory/factories/pydantic_factory.py:471: in build
    processed_kwargs = cls.process_kwargs(**kwargs)
.venv/lib/python3.11/site-packages/polyfactory/factories/base.py:986: in process_kwargs
    for field_meta in cls.get_model_fields():
.venv/lib/python3.11/site-packages/polyfactory/factories/pydantic_factory.py:417: in get_model_fields
    cls._fields_metadata = [
.venv/lib/python3.11/site-packages/polyfactory/factories/pydantic_factory.py:418: in <listcomp>
    PydanticFieldMeta.from_field_info(
.venv/lib/python3.11/site-packages/polyfactory/factories/pydantic_factory.py:215: in from_field_info
    return PydanticFieldMeta.from_type(
.venv/lib/python3.11/site-packages/polyfactory/field_meta.py:153: in from_type
    if field.type_args and not field.children:
.venv/lib/python3.11/site-packages/polyfactory/field_meta.py:96: in type_args
    return tuple(TYPE_MAPPING.get(arg, arg) for arg in get_args(self.annotation))
.venv/lib/python3.11/site-packages/polyfactory/field_meta.py:96: in <genexpr>
    return tuple(TYPE_MAPPING.get(arg, arg) for arg in get_args(self.annotation))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def __hash__(self):
>       return hash((self.__origin__, self.__metadata__))
E       TypeError: unhashable type: 'dict'

Neither does your first proposed approach, changing Bound=Hashable to Bound=List (at least, that's how I interpreted it).

I'll do some more digging; hints are welcome!

[adhtruong]

... so I hope Polyfactory won't consider this a remnant after its removal by Pydantic.

No plans to deprecate now as required to support other schema I believe eg. msgspec.

Neither does your first proposed approach, changing Bound=Hashable to Bound=List (at least, that's how I interpreted it).

I'll do some more digging; hints are welcome!

The bound here should be fine as refers to the item itself.

This hashability should already be fixed on main by #643. Does it work if try installing off main?

Will prepare a release soon

[WilliamDEdwards]

I get the same exception with #643, but I must admit I have not debugged it at all.