Prepare outlines to use outlines-core v0.2

[torymur]

outlines-core v0.2 introduces major interface changes, which require additional work on the outlines side.

Currently in outlines we're importing these from outlines_core:

from outlines_core.fsm.guide import Generate
from outlines_core.fsm.guide import Guide as CoreGuide
from outlines_core.fsm.guide import RegexGuide as CoreRegexGuide
from outlines_core.fsm.guide import Write
from outlines_core.fsm.guide import (
    create_states_mapping as uncached_create_states_mapping,
)

Now we're providing interfaces and usage like:

import json

from outlines_core.json_schema import build_regex_from_schema
from outlines_core.fsm.guide import Guide, Index, Vocabulary

schema =  {
  "title": "Foo",
  "type": "object",
  "properties": {"date": {"type": "string", "format": "date"}}
}
regex = build_regex_from_schema(json.dumps(schema))

vocabulary = Vocabulary.from_pretrained("openai-community/gpt2")
index = Index(regex, vocabulary)
guide = Guide(index)

# Get current state of the Guide:
current_state = guide.get_state()

# Get allowed tokens for the current state of the Guide:
allowed_tokens = guide.get_tokens()

# Advance Guide to the next state via some token_id and return allowed tokens for that new state:
next_allowed_tokens = guide.advance(allowed_tokens[-1])

# To check if Guide is finished:
guide.is_finished()

# If it's finished then this assertion holds:
assert guide.get_tokens() == [vocabulary.get_eos_token_id()]

More interface details for Vocabulary, Index and Guide will be available at: https://github.com/dottxt-ai/outlines-core/blob/main/python/outlines_core/outlines_core_rs.pyi

Some previously imported structures like Write and Generate might make sense to move back to outlines and adapt them, if they'll be used in other Guides. How best to proceed there is open for discussion.