Drupal is able to automatically discover the files that it needs to read. It does this by scanning known directories such as module and theme directories for files that have known suffixes such as .module, .info and others.
As you can imagine, the larger your code base, the more resource this process consumes. Thankfully Drupal is intelligent enough to perform this scan only when required, not on every request. At least that is how the system is supposed to work.
It is possible for your installation to become a little untidy in such a way that Drupal will come to conclusion that it does indeed need to scan your filesystem on every request, which can add seconds to every page load. This tends to happen when Drupal developer's have tried to execute a good house keeping plan and remove modules that are no longer required, which I think is a very good plan.
However, if you just delete the modules without uninstalling them correctly you will end up with an entry in the "system" database table that stil refers to your module. This entry tells Drupal your module still exists and so it thinks it needs to load the files for it, so it scans the filesystem for them. Because it is never able to find these files it is never satisfied and so will repeat this process on every page request.
All you need to do to fix this is remove the offending entry from you "system" database table.