Pitfalls of trying to mix Entity Framework providers for SQLite and SQL Server

As part of my annual evaluation, I was given the task of figuring out how we might use an in memory database (such as SQLite) to run integration tests against instead of using the full test database (SQL Server).  I was initially optimistic, but after some experimentation I have decided that this approach is fundamentally flawed.  While I think you could theoretically pull it off, you would never recoup the amount of work involved.  Ultimately, I think for integration tests you are better off just pointing EF at a test database of the same type.