Phase 3: Nested Mutation Tracking Demo¶

This notebook demonstrates Phase 3 features: transparent mutation tracking for nested data structures.

What's New in Phase 3?¶

• ProxiedMap: Transparent dictionary proxy that tracks all mutations
• ProxiedList: Transparent list proxy that tracks all mutations
• Firestore Constraints: Runtime validation of field names and nesting depth
• Conservative Saving: Entire fields saved when nested values change
• Automatic: Works for both assigned and fetched data

Setup¶

In [1]:

# Standard imports
from fire_prox.firestore_constraints import FirestoreConstraintError
from fire_prox.proxied_list import ProxiedList
from fire_prox.proxied_map import ProxiedMap

from fire_prox import FireProx
from fire_prox.testing import demo_client

# Initialize with demo client (connects to emulator)
client = demo_client()
db = FireProx(client)

# Get a test collection
users = db.collection('phase3_demo_users')

print("✅ Connected to Firestore emulator")

# Standard imports from fire_prox.firestore_constraints import FirestoreConstraintError from fire_prox.proxied_list import ProxiedList from fire_prox.proxied_map import ProxiedMap from fire_prox import FireProx from fire_prox.testing import demo_client # Initialize with demo client (connects to emulator) client = demo_client() db = FireProx(client) # Get a test collection users = db.collection('phase3_demo_users') print("✅ Connected to Firestore emulator")

✅ Connected to Firestore emulator

1. Basic Nested Dictionary Tracking¶

In Phase 1 and 2, nested mutations were not tracked:

In [2]:

# Create a user with nested settings
user = users.new()
user.name = 'Ada Lovelace'
user.settings = {
    'theme': 'dark',
    'fontSize': 14,
    'notifications': {
        'email': True,
        'sms': False,
        'push': True
    }
}
user.save()

print(f"✅ Created user: {user.id}")
print(f"   Name: {user.name}")
print(f"   Settings type: {type(user.settings).__name__}")
print(f"   Settings: {dict(user.settings)}")

# Create a user with nested settings user = users.new() user.name = 'Ada Lovelace' user.settings = { 'theme': 'dark', 'fontSize': 14, 'notifications': { 'email': True, 'sms': False, 'push': True } } user.save() print(f"✅ Created user: {user.id}") print(f" Name: {user.name}") print(f" Settings type: {type(user.settings).__name__}") print(f" Settings: {dict(user.settings)}")

✅ Created user: kqNvPmqOaDfwLP6FsY8K
   Name: Ada Lovelace
   Settings type: ProxiedMap
   Settings: {'theme': 'dark', 'fontSize': 14, 'notifications': ProxiedMap({'email': True, 'sms': False, 'push': True})}

In [3]:

# ✅ Phase 3: Nested mutations are automatically tracked!
print("Before mutation:")
print(f"  Is dirty? {user.is_dirty()}")
print(f"  Dirty fields: {user.dirty_fields}")

# Mutate nested value
user.settings['theme'] = 'light'

print("\nAfter mutation:")
print(f"  Is dirty? {user.is_dirty()}")
print(f"  Dirty fields: {user.dirty_fields}")
print(f"  Theme is now: {user.settings['theme']}")

# Save and verify
user.save()
print("\n✅ Saved! Dirty tracking cleared.")
print(f"  Is dirty? {user.is_dirty()}")

# ✅ Phase 3: Nested mutations are automatically tracked! print("Before mutation:") print(f" Is dirty? {user.is_dirty()}") print(f" Dirty fields: {user.dirty_fields}") # Mutate nested value user.settings['theme'] = 'light' print("\nAfter mutation:") print(f" Is dirty? {user.is_dirty()}") print(f" Dirty fields: {user.dirty_fields}") print(f" Theme is now: {user.settings['theme']}") # Save and verify user.save() print("\n✅ Saved! Dirty tracking cleared.") print(f" Is dirty? {user.is_dirty()}")

Before mutation:
  Is dirty? False
  Dirty fields: set()

After mutation:
  Is dirty? True
  Dirty fields: {'settings'}
  Theme is now: light

✅ Saved! Dirty tracking cleared.
  Is dirty? False

In [4]:

# Verify persistence: fetch fresh copy
user2 = users.doc(user.id)
user2.fetch()

print("Fresh copy from Firestore:")
print(f"  Theme: {user2.settings['theme']}")
print("  ✅ Mutation was persisted!")

# Verify persistence: fetch fresh copy user2 = users.doc(user.id) user2.fetch() print("Fresh copy from Firestore:") print(f" Theme: {user2.settings['theme']}") print(" ✅ Mutation was persisted!")

Fresh copy from Firestore:
  Theme: light
  ✅ Mutation was persisted!

2. Deeply Nested Mutation Tracking¶

Proxies work recursively at any depth:

In [5]:

# Create deeply nested structure
user = users.new()
user.name = 'Grace Hopper'
user.config = {
    'ui': {
        'theme': {
            'colors': {
                'primary': '#ff0000',
                'secondary': '#00ff00',
                'accent': '#0000ff'
            }
        },
        'layout': 'grid'
    },
    'performance': {
        'caching': True
    }
}
user.save()

print(f"✅ Created user with 4-level nesting: {user.id}")

# Create deeply nested structure user = users.new() user.name = 'Grace Hopper' user.config = { 'ui': { 'theme': { 'colors': { 'primary': '#ff0000', 'secondary': '#00ff00', 'accent': '#0000ff' } }, 'layout': 'grid' }, 'performance': { 'caching': True } } user.save() print(f"✅ Created user with 4-level nesting: {user.id}")

✅ Created user with 4-level nesting: 7unS8HR8WhrodK9p9Pue

In [6]:

# ✅ Mutate deeply nested value (4 levels deep)
print(f"Before: primary color = {user.config['ui']['theme']['colors']['primary']}")

user.config['ui']['theme']['colors']['primary'] = '#9900ff'

print(f"After: primary color = {user.config['ui']['theme']['colors']['primary']}")
print(f"Is dirty? {user.is_dirty()}")
print(f"Dirty fields: {user.dirty_fields}")

# Save and verify
user.save()
user.fetch(force=True)
print(f"\n✅ Persisted: {user.config['ui']['theme']['colors']['primary']}")

# ✅ Mutate deeply nested value (4 levels deep) print(f"Before: primary color = {user.config['ui']['theme']['colors']['primary']}") user.config['ui']['theme']['colors']['primary'] = '#9900ff' print(f"After: primary color = {user.config['ui']['theme']['colors']['primary']}") print(f"Is dirty? {user.is_dirty()}") print(f"Dirty fields: {user.dirty_fields}") # Save and verify user.save() user.fetch(force=True) print(f"\n✅ Persisted: {user.config['ui']['theme']['colors']['primary']}")

Before: primary color = #ff0000
After: primary color = #9900ff
Is dirty? True
Dirty fields: {'config'}

✅ Persisted: #9900ff

3. List Mutation Tracking¶

Lists are also wrapped in transparent proxies:

In [7]:

# Create user with tags list
user = users.new()
user.name = 'Alan Turing'
user.tags = ['mathematics', 'cryptography', 'ai']
user.save()

print(f"✅ Created user: {user.id}")
print(f"   Tags type: {type(user.tags).__name__}")
print(f"   Tags: {list(user.tags)}")

# Create user with tags list user = users.new() user.name = 'Alan Turing' user.tags = ['mathematics', 'cryptography', 'ai'] user.save() print(f"✅ Created user: {user.id}") print(f" Tags type: {type(user.tags).__name__}") print(f" Tags: {list(user.tags)}")

✅ Created user: API64qpqwuZRnVk10b3v
   Tags type: ProxiedList
   Tags: ['mathematics', 'cryptography', 'ai']

In [8]:

# ✅ List mutations tracked
print("Testing various list operations:\n")

# Append
user.tags.append('computing')
print(f"After append: {list(user.tags)}")
print(f"Is dirty? {user.is_dirty()}")
user.save()

# Extend
user.tags.extend(['logic', 'philosophy'])
print(f"\nAfter extend: {list(user.tags)}")
print(f"Is dirty? {user.is_dirty()}")
user.save()

# Index assignment
user.tags[0] = 'pure-mathematics'
print(f"\nAfter index assignment: {list(user.tags)}")
print(f"Is dirty? {user.is_dirty()}")
user.save()

# Remove
user.tags.remove('philosophy')
print(f"\nAfter remove: {list(user.tags)}")
print(f"Is dirty? {user.is_dirty()}")
user.save()

print("\n✅ All list operations tracked and persisted!")

# ✅ List mutations tracked print("Testing various list operations:\n") # Append user.tags.append('computing') print(f"After append: {list(user.tags)}") print(f"Is dirty? {user.is_dirty()}") user.save() # Extend user.tags.extend(['logic', 'philosophy']) print(f"\nAfter extend: {list(user.tags)}") print(f"Is dirty? {user.is_dirty()}") user.save() # Index assignment user.tags[0] = 'pure-mathematics' print(f"\nAfter index assignment: {list(user.tags)}") print(f"Is dirty? {user.is_dirty()}") user.save() # Remove user.tags.remove('philosophy') print(f"\nAfter remove: {list(user.tags)}") print(f"Is dirty? {user.is_dirty()}") user.save() print("\n✅ All list operations tracked and persisted!")

Testing various list operations:

After append: ['mathematics', 'cryptography', 'ai', 'computing']
Is dirty? True

After extend: ['mathematics', 'cryptography', 'ai', 'computing', 'logic', 'philosophy']
Is dirty? True

After index assignment: ['pure-mathematics', 'cryptography', 'ai', 'computing', 'logic', 'philosophy']
Is dirty? True

After remove: ['pure-mathematics', 'cryptography', 'ai', 'computing', 'logic']
Is dirty? True

✅ All list operations tracked and persisted!

4. Mixed Nested Structures¶

Proxies handle complex combinations of lists and dicts:

In [9]:

# Lists within dicts, dicts within lists
user = users.new()
user.name = 'Katherine Johnson'
user.data = {
    'projects': [
        {'name': 'Apollo 11', 'year': 1969, 'role': 'mathematician'},
        {'name': 'Mercury', 'year': 1961, 'role': 'computer'}
    ],
    'achievements': {
        'awards': ['Presidential Medal of Freedom', 'Congressional Gold Medal'],
        'publications': 26
    }
}
user.save()

print(f"✅ Created user with mixed nested structures: {user.id}")

# Lists within dicts, dicts within lists user = users.new() user.name = 'Katherine Johnson' user.data = { 'projects': [ {'name': 'Apollo 11', 'year': 1969, 'role': 'mathematician'}, {'name': 'Mercury', 'year': 1961, 'role': 'computer'} ], 'achievements': { 'awards': ['Presidential Medal of Freedom', 'Congressional Gold Medal'], 'publications': 26 } } user.save() print(f"✅ Created user with mixed nested structures: {user.id}")

✅ Created user with mixed nested structures: YjoSX8xvM3ON3KMRj1Hr

In [10]:

# ✅ Mutate dict within list
print("Before: Apollo role =", user.data['projects'][0]['role'])
user.data['projects'][0]['role'] = 'lead-mathematician'
print("After: Apollo role =", user.data['projects'][0]['role'])
print(f"Is dirty? {user.is_dirty()}\n")
user.save()

# ✅ Mutate list within dict
print("Before: awards =", list(user.data['achievements']['awards']))
user.data['achievements']['awards'].append('NACA Outstanding Achievement Medal')
print("After: awards =", list(user.data['achievements']['awards']))
print(f"Is dirty? {user.is_dirty()}")
user.save()

print("\n✅ Mixed nested mutations tracked!")

# ✅ Mutate dict within list print("Before: Apollo role =", user.data['projects'][0]['role']) user.data['projects'][0]['role'] = 'lead-mathematician' print("After: Apollo role =", user.data['projects'][0]['role']) print(f"Is dirty? {user.is_dirty()}\n") user.save() # ✅ Mutate list within dict print("Before: awards =", list(user.data['achievements']['awards'])) user.data['achievements']['awards'].append('NACA Outstanding Achievement Medal') print("After: awards =", list(user.data['achievements']['awards'])) print(f"Is dirty? {user.is_dirty()}") user.save() print("\n✅ Mixed nested mutations tracked!")

Before: Apollo role = mathematician
After: Apollo role = lead-mathematician
Is dirty? True

Before: awards = ['Presidential Medal of Freedom', 'Congressional Gold Medal']
After: awards = ['Presidential Medal of Freedom', 'Congressional Gold Medal', 'NACA Outstanding Achievement Medal']
Is dirty? True

✅ Mixed nested mutations tracked!

5. Firestore Constraint Validation¶

Phase 3 enforces Firestore's documented constraints at assignment time:

In [11]:

# ❌ Invalid field name: reserved __name__ pattern
user = users.new()
user.name = 'Test User'

try:
    user.settings = {'__invalid__': 'value'}
except FirestoreConstraintError as e:
    print("❌ Caught constraint error:")
    print(f"   {e}")
    print("")

# ❌ Field name with whitespace
try:
    user.settings = {'field name': 'value'}
except FirestoreConstraintError as e:
    print("❌ Caught constraint error:")
    print(f"   {e}")
    print("")

# ✅ Valid field names work fine
user.settings = {'valid_field': 'value', 'another_field': 123}
print("✅ Valid field names accepted")

# ❌ Invalid field name: reserved __name__ pattern user = users.new() user.name = 'Test User' try: user.settings = {'__invalid__': 'value'} except FirestoreConstraintError as e: print("❌ Caught constraint error:") print(f" {e}") print("") # ❌ Field name with whitespace try: user.settings = {'field name': 'value'} except FirestoreConstraintError as e: print("❌ Caught constraint error:") print(f" {e}") print("") # ✅ Valid field names work fine user.settings = {'valid_field': 'value', 'another_field': 123} print("✅ Valid field names accepted")

❌ Caught constraint error:
   Field name '__invalid__' cannot match __name__ pattern (at depth 0). Firestore reserves double-underscore names for internal use.

✅ Valid field names accepted

In [12]:

# ❌ Excessive nesting depth (Firestore limit: 20 levels)
user = users.new()

# Build a structure exceeding 20 levels
data = {'level': {}}
current = data['level']
for i in range(25):
    current['level'] = {}
    current = current['level']

try:
    user.data = data
except FirestoreConstraintError as e:
    print("❌ Caught nesting depth error:")
    print(f"   {e}")
    print("")

# ✅ Reasonable nesting works fine
user.data = {'a': {'b': {'c': {'d': 'value'}}}}
print("✅ Reasonable nesting (4 levels) accepted")

# ❌ Excessive nesting depth (Firestore limit: 20 levels) user = users.new() # Build a structure exceeding 20 levels data = {'level': {}} current = data['level'] for i in range(25): current['level'] = {} current = current['level'] try: user.data = data except FirestoreConstraintError as e: print("❌ Caught nesting depth error:") print(f" {e}") print("") # ✅ Reasonable nesting works fine user.data = {'a': {'b': {'c': {'d': 'value'}}}} print("✅ Reasonable nesting (4 levels) accepted")

❌ Caught nesting depth error:
   Firestore nesting depth limit exceeded at path 'data'. Maximum depth is 20 levels, attempted 21. Consider flattening your data structure or using subcollections.

✅ Reasonable nesting (4 levels) accepted

6. Conservative Saving¶

When a nested value changes, the entire top-level field is saved:

In [13]:

# Create user with large nested structure
user = users.new()
user.name = 'Demo User'
user.config = {
    'theme': 'dark',
    'fontSize': 14,
    'language': 'en',
    'nested': {
        'value1': 'old',
        'value2': 'unchanged'
    }
}
user.save()

print("Initial config:")
print(dict(user.config))

# Create user with large nested structure user = users.new() user.name = 'Demo User' user.config = { 'theme': 'dark', 'fontSize': 14, 'language': 'en', 'nested': { 'value1': 'old', 'value2': 'unchanged' } } user.save() print("Initial config:") print(dict(user.config))

Initial config:
{'theme': 'dark', 'fontSize': 14, 'language': 'en', 'nested': ProxiedMap({'value1': 'old', 'value2': 'unchanged'})}

In [14]:

# Change ONE nested value
user.config['nested']['value1'] = 'new'
print(f"\nChanged nested value, dirty fields: {user.dirty_fields}")

# Save (writes entire 'config' field)
user.save()

# Verify: fetch fresh copy
user2 = users.doc(user.id)
user2.fetch()

print("\nFresh copy from Firestore:")
print(f"  theme: {user2.config['theme']} (unchanged)")
print(f"  fontSize: {user2.config['fontSize']} (unchanged)")
print(f"  nested.value1: {user2.config['nested']['value1']} (changed)")
print(f"  nested.value2: {user2.config['nested']['value2']} (unchanged)")
print("\n✅ Entire field saved, all values present")

# Change ONE nested value user.config['nested']['value1'] = 'new' print(f"\nChanged nested value, dirty fields: {user.dirty_fields}") # Save (writes entire 'config' field) user.save() # Verify: fetch fresh copy user2 = users.doc(user.id) user2.fetch() print("\nFresh copy from Firestore:") print(f" theme: {user2.config['theme']} (unchanged)") print(f" fontSize: {user2.config['fontSize']} (unchanged)") print(f" nested.value1: {user2.config['nested']['value1']} (changed)") print(f" nested.value2: {user2.config['nested']['value2']} (unchanged)") print("\n✅ Entire field saved, all values present")

Changed nested value, dirty fields: {'config'}

Fresh copy from Firestore:
  theme: dark (unchanged)
  fontSize: 14 (unchanged)
  nested.value1: new (changed)
  nested.value2: unchanged (unchanged)

✅ Entire field saved, all values present

7. Fetched Data is Wrapped¶

Data fetched from Firestore is automatically wrapped in proxies:

In [15]:

# Create using native client (bypassing FireProx)

native_ref = client.collection('phase3_demo_users').document()
native_ref.set({
    'name': 'Native User',
    'settings': {'theme': 'dark', 'fontSize': 12},
    'tags': ['python', 'firestore']
})

print(f"✅ Created via native client: {native_ref.id}")

# Create using native client (bypassing FireProx) native_ref = client.collection('phase3_demo_users').document() native_ref.set({ 'name': 'Native User', 'settings': {'theme': 'dark', 'fontSize': 12}, 'tags': ['python', 'firestore'] }) print(f"✅ Created via native client: {native_ref.id}")

✅ Created via native client: DUJjWrAX9jq3WAJntd9p

In [16]:

# Fetch via FireProx
user = users.doc(native_ref.id)
user.fetch()

print("Fetched data types:")
print(f"  settings: {type(user.settings).__name__}")
print(f"  tags: {type(user.tags).__name__}")
print(f"  Is ProxiedMap? {isinstance(user.settings, ProxiedMap)}")
print(f"  Is ProxiedList? {isinstance(user.tags, ProxiedList)}")

# ✅ Mutations on fetched data are tracked
user.settings['theme'] = 'light'
print(f"\n✅ Mutation tracked: {user.is_dirty()}")

# Fetch via FireProx user = users.doc(native_ref.id) user.fetch() print("Fetched data types:") print(f" settings: {type(user.settings).__name__}") print(f" tags: {type(user.tags).__name__}") print(f" Is ProxiedMap? {isinstance(user.settings, ProxiedMap)}") print(f" Is ProxiedList? {isinstance(user.tags, ProxiedList)}") # ✅ Mutations on fetched data are tracked user.settings['theme'] = 'light' print(f"\n✅ Mutation tracked: {user.is_dirty()}")

Fetched data types:
  settings: ProxiedMap
  tags: ProxiedList
  Is ProxiedMap? True
  Is ProxiedList? True

✅ Mutation tracked: True

8. to_dict() Returns Plain Types¶

For user-facing output, to_dict() unwraps proxies:

In [17]:

user = users.new()
user.name = 'Export Test'
user.settings = {'theme': 'dark'}
user.tags = ['test']
user.save()

# Internal storage uses proxies
print("Internal types:")
print(f"  settings: {type(user.settings).__name__}")
print(f"  tags: {type(user.tags).__name__}")

# to_dict() returns plain types
data = user.to_dict()
print("\nto_dict() types:")
print(f"  settings: {type(data['settings']).__name__}")
print(f"  tags: {type(data['tags']).__name__}")
print("\n✅ Proxies unwrapped to plain dict/list")

user = users.new() user.name = 'Export Test' user.settings = {'theme': 'dark'} user.tags = ['test'] user.save() # Internal storage uses proxies print("Internal types:") print(f" settings: {type(user.settings).__name__}") print(f" tags: {type(user.tags).__name__}") # to_dict() returns plain types data = user.to_dict() print("\nto_dict() types:") print(f" settings: {type(data['settings']).__name__}") print(f" tags: {type(data['tags']).__name__}") print("\n✅ Proxies unwrapped to plain dict/list")

Internal types:
  settings: ProxiedMap
  tags: ProxiedList

to_dict() types:
  settings: dict
  tags: list

✅ Proxies unwrapped to plain dict/list

9. Async API¶

Everything works identically with the async API:

In [18]:

from fire_prox import AsyncFireProx
from fire_prox.testing import async_demo_client

# Initialize async client
async_client = async_demo_client()
async_db = AsyncFireProx(async_client)
async_users = async_db.collection('phase3_demo_async_users')

print("✅ Initialized async client")

from fire_prox import AsyncFireProx from fire_prox.testing import async_demo_client # Initialize async client async_client = async_demo_client() async_db = AsyncFireProx(async_client) async_users = async_db.collection('phase3_demo_async_users') print("✅ Initialized async client")

✅ Initialized async client

In [19]:

# Async nested mutation tracking
user = async_users.new()
user.name = 'Async User'
user.settings = {'theme': 'dark', 'notifications': {'email': True}}
await user.save()

print(f"✅ Created async user: {user.id}")
print(f"   Settings type: {type(user.settings).__name__}")

# Mutate nested value
user.settings['theme'] = 'light'
print(f"   Is dirty? {user.is_dirty()}")

await user.save()
print("   ✅ Saved with await")

# Fetch and verify
user2 = async_users.doc(user.id)
await user2.fetch()
print(f"   Theme persisted: {user2.settings['theme']}")

# Async nested mutation tracking user = async_users.new() user.name = 'Async User' user.settings = {'theme': 'dark', 'notifications': {'email': True}} await user.save() print(f"✅ Created async user: {user.id}") print(f" Settings type: {type(user.settings).__name__}") # Mutate nested value user.settings['theme'] = 'light' print(f" Is dirty? {user.is_dirty()}") await user.save() print(" ✅ Saved with await") # Fetch and verify user2 = async_users.doc(user.id) await user2.fetch() print(f" Theme persisted: {user2.settings['theme']}")

✅ Created async user: m0Vl1zJUEtsGPim9iHJC
   Settings type: ProxiedMap
   Is dirty? True
   ✅ Saved with await
   Theme persisted: light

Summary¶

Phase 3 delivers:

✅ Transparent Tracking: Nested mutations detected automatically

✅ Firestore Constraints: Field names and nesting depth validated at assignment

✅ Conservative Saving: Entire fields saved for data integrity

✅ Natural API: Proxies behave exactly like native dicts and lists

✅ Both APIs: Full sync and async support

✅ Zero Breaking Changes: All existing code continues to work

Next Steps¶

• Explore the Phase 3 Implementation Report
• Check out the test suite
• Review the Architectural Blueprint