With the proliferation of data-hungry accelerators and co-processors in embedded system design, co-design of processors and memory systems is becoming more important. Current simulation techniques for processors rely on oversimplified and inflexible memory models, while techniques for memory system simulation tend to only utilize simple processor models. In this work, we integrate a cycle-accurate Verilator processor and vector co-processor model with the Gem5 memory simulator in order to evaluate the full impact of a data-hungry co-processor on the memory system and main core performance, and to provide a framework for future co-design of both processor and memory systems.