Since the Speaker of the House of Representatives sets the agenda, is there any way to unseat the Speaker ?

Curiously, there's nothing in the Constitution that requires the House to elect one of its own members as a Speaker (Article I, Section 26), so defeating a reigning Speaker in his/her district isn't necessarily a way to get him/her out.

According to the Rules of the House (which are based on Jefferson's Manual), a Speaker may be removed at the will of the House, and a Speaker pro tempore appointed. (Rules of the House, Sec. 315.)  The annotations to the Manual provided by the Office of the Clerk of the House note that the House has never removed a Speaker (although they have removed other sitting officers).  A resolution to remove an officer is a matter of privilege, meaning it has precedence over all other questions except a motion to adjourn. (Rules, Sec. 664 et seq.)

The "question of privilege" applies to your question, in that the Speaker cannot elect not to address a question of privilege, even if it involves him/herself being unseated.  The Clerk's commentary on the handling of questions of privilege notes that, "while under Rule IX a question of privileges of the House takes precedence over all other questions," the Speaker "may entertain unanimous consent requests for 'one-minute speeches' pending recognition for a question of privilege, since such requests . . . temporarily waive the standing rules of the House relating to the order of business" (Rules of the House, sec. 665).  The practical outcome of such a move would be a House-equivalent of a Senate filibuster -- if the Speaker can keep those one-minutes coming, he/she can postpone impending doom while his/her allies are out rounding up allies.

Informally, my colleagues are right -- unseating a sitting Speaker by defeating him/her in a district election is the most common way to achieve "involunary retirement."  You might find it interesting to read up on the life and times of Czar (or Uncle) Joe Cannon, who was a master at Speakership (1903-1911) and dominated the House, but was finally ousted by a change in the make-up of the Congress from Republican to Democrat.  By the way, that's another way to get a new Speaker -- change which party constitutes the majority in the House.

Other less direct means of getting him/her out is to promote the incumbent to Vice President or some other government position (which requires the Speaker to resign his/her position) or remove both the President and Vice President from office -- which would elevate the Speaker to the Presidency, again requiring resignation from other government positions.

So, the short answer to your question is, yes, there are ways to unseat the Speaker.