Checkoff 03: Camera II
MIT Fall 2024
Please Log In for full access to the web site.
Note that this link will take you to an external site (https://shimmer.mit.edu) to authenticate, and then you will be redirected back to this page.
You are not to remove a camera module from lab. We do not have enough for everyone to have one simultaneously. They need to be shared. If we catch you taking one from lab space, there will be consequences. When you are not working with the camera, please return it to the front of the lab area into its area.
Pipelines
Your center of mass should be working and you should be tracking things, but the system is also horribly pipelined at this point. The propagation of information via various routes through the system get delayed differing amounts of clock cycles resulting in data from one point in time getting used with data from another point in time. It is awful. A very obvious way in which this should manifest is that you'll see a large bar of pixels on the left side of your screen that are actually from the right side of the screen. This is because we haven't balanced the hsync and vsync signals with the pixel data. Another, less obvious issue is that as we've discussed in lectures 08 and 09, accessing information for an image on our FPGA takes four clock cycles. Consequently when rendering your Image Sprite, you're going to see an artifact of that in the top left (of the sprite) consisting of several white pixels; depending on your implementation, your edges may also have problem. You should make sure that you properly pipeline your entire system to account for these delays. What signals will this require you to pipeline? The crosshair and the RGB to YCrCb conversion also have their own problems.
The input to horizontal sync and vertical sync signals aren't the only path that has problems. The input signals to the center of mass are also poorly pipelined at this point meaning you'll be counting the wrong pixel locations. This also needs to be fixed.
How is a system properly pipelined? By determining all the locations of latencies and making sure that other signals are properly delayed so that any downstream circuits receive data all related to the same timestep. The system already has a simple example of pipelining. Consider around the frame_buffer module. The up-scaling logic provides two output values, addrb which is the address of the pixel we want to look up from the frame buffer and a good_addrb signal which is used to determine if the values on addrb are valid given the bounds of the up-scaling. Because frame_buffer is a standard HIGH_PERFORMANCE Xilinx BRAM with a two-clock-cycle latency, that means when we consider whether or not the value of the pixel is legitimate as indicated by good_addb we can't just use that signal as it is since it is referring to a different timestep.
To fix this we need to delay it by two clock cycles and use that value. As a result, when we make the comparison, and passing on frame_buff_raw or 0's we're doing it with information all related to the same time step.
Make sure to update your HDL source to reflect this change.
Now...
You need to apply this same sort of logic to the entire system. The state of your pipelining is shown in the schematic below (this does not necessarily contain all signals...only ones relevant to the pipelining problem). As just stated, there are two pipelines we currently have to worry about balancing: (start-to-endpoint1 and start-to-endpoint2). These both bust be made into well-balanced, k- pipelines. While they are separate, they are not independent, based on how the signals are used in both.
What we would like to do is balance both pipelines so each Start-to-Endpoint is well-balanced in terms of latency. This can be accomplished by placing the correct amount of registers into the nine locations shown below. Study the figure, and determine the nine numbers.
While there are infinite solutions that will result in a well balanced, we want the solution that has the least amount of latency overall. This results in a balance, but the least delay (nobody wants delay, and also the least valid delay uses the fewest additional resources...always a good goal to strive for.). Also note this is not the only way to pipeline this system (we'll try another one down below). On your final project, you can pipeline however you want (as long as it is correct).
In the nine boxes below, enter the correct number of registers needed to balance both pipelines.
Retiming
Now consider this slightly re-piped configuration. Note all we did here was tap the signals for hcount and vcount earlier for a few modules (the image sprite and crosshair).
In the nine boxes below, enter the correct number of registers needed to balance both pipelines.
Build It!
OK we now have two options (you pick one) we know where and how to pipeline our system, we need to implement it in Verilog. The easiest way to do this will be to create a series of appropriately-sized unpacked array for the signals that need to be pipelined. For example, hcount definitely needs pipelining at some point (retroactive hint). We could accomplish creating of a pipeline with the following (which could be implemented in top_level):
//vary the packed width based on signal
//vary the unpacked width based on pipelining depth needed
logic [10:0] hcount_pipe [STAGES_NEEDED-1:0];
always_ff @(posedge clk_pixel)begin
hcount_pipe[0] <= hcount;
for (int i=1; i<STAGES_NEEDED; i = i+1)begin
hcount_pipe[i] <= hcount_pipe[i-1];
end
end
Then replace the appropriate instances of hcount in your code. For your aid/convenience they have been marked with a //TODO... line with some notes, but make sure you are double-checking everything1. For example in the image_sprite, you'll see this line:
.hcount_in(hcount), //TODO: needs to use pipelined signal (PS3 or None depending on choice)
You'd want to replace this with something like:
.hcount_in(hcount_piped[BLAH]), //BLAH corresponds to your pipelined value for PS1!
You could also write a module to handle pipelining for you kinda like what we did in lecture.
Another location where a pipelining mismatch will occur is likely inside your image_sprite module. The logic that "releases" the image data is not synced in time with the data coming from the BRAM. So fix it!
When you've completed this pipelining, your system should still work as before, but slightly better! You should see significantly less "wrong" stuff on the left side of the screen, for example. There should also NOT be an artifact where your image sprite is being drawn (upper left corner white pixels). If you're still seeing that..nope...better fix it.
Show staff how you pipelined and what signals you pipelined.