In many cases, the interest rate on their accounts has been 0.5% or lower - the same as the Bank of England's base rate.
Savers should be given clearer information and helped to switch to better accounts, said the FCA.
The banking industry said the low interest rate environment was to blame.
The FCA said that 80% of easy access accounts had not been switched in the past three years.
Furthermore, those who have older savings accounts tend to earn less interest than those who have opened them more recently.
Despite its criticisms, the City regulator stopped short of banning bonus rates. Such rates operate for a limited period, and are used to tempt savers in.
Analysis: Simon Gompertz, BBC personal finance correspondent
The financial watchdog is barking and growling at banks over the way they let down customers over savings rates.
In too many cases, the saver is left languishing on a pitiful rate, given little information on how bad it is, and seldom encouraged to shop around for something better.
So the FCA has devised a shiny new weapon to cow savings providers into behaving better. When they write to you with a regular statement, they will have to include a "switching box" which shows in a colourful chart how bad your rate is, how much more you could get from the provider's other accounts, and what you could earn from the average of the best 10 savings accounts on the market.
Another innovation is that when you get the account in the first place, they will have to include a warning if the rate is a poor one, saying it pays "interest below the Bank of England's base rate".
The watchdog has retreated with a whine and a lowered tail from drastic measures such as banning the introductory bonus rates that make accounts seem attractive at first, but leave you stuck on a measly rate later.
But the switching box could be a big embarrassment for banks and building societies that take advantage of gullible customers.
The big High Street banks - "large personal current account providers"- are singled out for particular criticism.
The FCA said they attracted the most money into easy access accounts, even though they offer the lowest interest rates.
"In a good market, firms should be competing to offer the best possible deal and consumers should have the information they need to help them shop around," said Christopher Woolard, the FCA's director of strategy and competition.
As a result of its investigation, the FCA is proposing:
- Providers should be more transparent about reductions in savings rates over time
- Savers should be told the lowest interest rate that any customers are getting on a particular account
- There should be clearer, more timely information to help savers compare, and then switch, their accounts
- Consumers should be able to view and manage different accounts in one place
- A reduction in the 15 day switching time for cash Individual Savings Accounts (ISAs).
In response, Lloyds Bank said it was already improving the simplicity and transparency of its savings accounts.
It promised to introduce just three accounts for each of its main brands - Halifax, Bank of Scotland and Lloyds.
The banking industry said it would consider the FCA's report carefully.
"These have been frustrating years for savers," said a spokesperson for the British Bankers Association (BBA).
"More than five years of the Bank of England's base rate at a record low has fostered a low interest rate environment and that can be hard."
The FCA will now consult on its plans, until 18 February.