This is the perfect type of issue to be addressed in a brief and/or initial client meetings. Exploring all the possibilities for how the design will be used helps define the project as well anticipate what file formats might be needed at its conclusion. The goal is to meet your clients’ needs and expectations but not necessarily exceed them. Overwhelming a client with files they might not understand or use is counterproductive. Besides being available to provide new files everyone once in awhile provides a great opportunity to reconnect.
So the nitty gritty… what is good for what? Always include a PDF, it is useful for myriad things especially printing. Next, a file for the web is almost always mandatory, so a JPEG or a PNG. Finally, an EPS is great for scaling and some vector based applications, like vinyl cutting. Those three files should cover most basic usage, perhaps throw in a monochromatic version if applicable. Anything beyond that would be dictated by the client’s specific needs. It is a good idea to annotate the files so that the preferred use is apparent.
Note: This advice is only useful for relatively small clients with needs you can predict. A larger client may need a more robust solution that might include dozens of files, with multiple lock-ups and permutations to preserve brand integrity. Providing as few files as possible would still be a best practice.